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We apply an information theoretic treatment of action potential time series measured with micro- 
electrode arrays to estimate the connectivity of mammalian neuronal cell assemblies grown in vitro. 
We infer connectivity between two neurons via the measurement of the mutual information between 
their spike trains. In addition we measure higher point multi-informations between any two spike 
trains conditional on the activity of a third cell, as a means to identify and distinguish classes of 
functional connectivity among three neurons. The use of a conditional three-cell measure removes 
some interpretational shortcomings of the pairwise mutual information and sheds light into the func- 
tional connectivity arrangements of any three cells. We analyze the resultant connectivity graphs 
in light of other complex networks and demonstrate that, despite their ex vivo development, the 
connectivity maps derived from cultured neural assemblies are similar to other biological networks 
and display nontrivial structure in clustering coefficient, network diameter and assortative mixing. 
Specifically we show that these networks are weakly disassortative small world graphs, which dif- 
fer significantly in their structure from randomized graphs with the same degree. We expect our 
analysis to be useful in identifying the computational motifs of a wide variety of complex networks, 
derived from time series data. 



I. INTRODUCTION 



Understanding and quantifying the dynamical mechanisms used by the nervous system to store and process infor- 
mation remains one of the greatest challenges to contemporary science. At present, broad outlines of the physical 
mechanisms that underpin the basic functioning of single neurons and synapses in the brain are understood 
However, this detailed knowledge of individual units sheds little light into the origin of the unmatched computational 
power of mammalian nervous systems, achieved despite characteristic operating times that are six orders of magnitude 
slower than those of modern digital computers. 

The computational nature of the brain lies therefore principally in the ensemble properties of neurons, synapses 
and their emergent complex, dynamical networks. Over the last few years the interaction structure of many complex 
systems has been mapped in terms of graphs, which can in turn be characterized using tools of statistical physics (2j. 
This approach has revealed broad classes of networks such as small world graphs [3| and scale free networks ,4| , which 
occur across fields of study, from technological networks, such as the internet, to various biological and social systems. 
The structural properties of these graphs, such as their degree distribution or their local transitivity, moreover, have 
been suggested to result from optimization constraints [g] or network growth dynamics @ , thus connecting graph 
structure to operational definitions of function, independent of a system's details. These lines of research provide 
new quantitative insights, connecting the interaction structure of a complex systems to novel definitions of function 
in fields where quantitative syntheses have been lacking. 

The mammalian nervous system is, arguably, simultaneously the most complex and fascinating of all networks. 
Studies that pursue its quantitative understanding in terms of structural, dynamical networks of simple units (neurons 
and synapses) 0] are now beginning to be possible. Notwithstanding recent reconstructions for invertebrate cells [H, 
which form relatively small numbers of connections to other cells, measuring the structure of living networks of 
neurons directly is a difficult problem due to typical huge synaptic densities (about 10 8 per mm 3 in cortex) and large 
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potential connectivity, with degree per cell of order 10 4 @. Much more accessible are electrophysiological signals - 
action potentials or 'spikes' - generated by neurons. The electro-physiological technology to measure these signals 
over hundreds or even thousands of individual cells is now mature. Other new techniques based on fluorescence of 
electrically active cells can also be used to map correlated activity, and putative connectivity (lp| . Because action 
potentials are the signals encoding information in the brain, the study of collective network activity in terms of 
their time series is both natural and important. From these types of data 'networks of information' can be inferred, 
revealing how signals are shared between neurons, stored and collectively processed. 

Here we construct networks of neurons, based on the information theoretic treatment of time series of action 
potentials from cortical neuronal networks grown in vitro. Though these systems may differ substantially from an 
intact brain they are interesting both as non-trivial starting points for in vivo analysis, and as examples of living 
neuronal networks capable of non-trivial computation [TT| . For these reasons neuronal networks in vitro have received 
much recent attention in terms of their collective dynamical and statistical properties [HI, [l3|, 0, HE, Oil Ell EM Gil • 

Below we develop a methodology, based on the information theoretic treatment of multiple cell spike time series, 
to map the structure of networks of neurons. We show that the resulting graph displays non-trivial structure shared 
by other complex networks. Our approach also identifies collective functional modules consisting of groups of neurons 
that enable different modes of information processing in nervous systems. 

The remaining of this paper is organized as follows. In section |TT] we give details of our experimental setting and 
of the electrophysiological activity recordings. We also introduce the several information theoretic quantities to be 
used subsequently to reconstruct a proxy connectivity network. We show how the use of several correlation and 
information theoretic quantities taken together can lead to the identification of specific connectivity structures, when 
multiple neurons are considered simultaneously. In section UlTl we give the results of these methods applied to data. 
Section IIVI analyzes the resulting graph in light of the properties of other complex networks. Finally we present the 
outlook for the methods proposed here, particularly in determining the dynamical evolution of networks and their 
collective response under driven conditions. In addition in Appendix A we prove a relation between definitions of 
information theoretic quantities, introduced in section IIIII 

II. EXPERIMENTS AND DATA 

We anal yze the time series of mammalian (murine) spiking cortical networks living in culture over microelectrode 
arrays [20L l2ll. |22T ] . These platforms, along with other types of neuronal culture [l5|, [Til Eil , currently allow the most 
extensive coverage of network elecrophysiological activity, in terms of both numbers of cells and length of recordings. 
Naturally, cultured cell networks have presumably a simpler structure than their counterparts in vivo. Nevertheless 
they are fully active neuronal networks, displaying many of the qualitative properties of developing mammalian tissue 
[Til ] . The topology of networks of neurons formed in vitro remains largely uncharted, although recent work in this 
direction has started to appear [1, E3] • 

Neuronal networks on microelectrode arrays (MEAs) are assemblies of mammalian nerve cells growing on fields 
of non-invasive, substrate-integrated microelectrodes, see Fig. [TJ The cells remain viable for over six months and 
form spontaneously active networks. Tissue is dissected from mouse embryos, dissociated, and then seeded on MEAs. 
Neural activity takes the form of action potentials (or spikes): short pulses (at which we record a time stamp) occurring 
when a neuronal membrane potential crosses threshold. Each channel (up to 4 per electrode, scanned at 40 KHz) is 
associated with a specific neuron via its action potential shape (spike sorting). Cell-electrode coupling is stable so 
that wave-shapes can be followed with accuracy, for many channels and over many days at a time. 

The results presented below were obtained from five hour recordings taken from three different cortical cultures, 
spanning a range of sizes, with 20, 33 and 62 recorded cells. The three networks are mature (see e.g. [l3[), with 
ages in vitro of 34, 51 and 42 days, respectively. The number of cells recorded are a fraction (estimated roughly at 
5-10%) of all active neurons in the culture. The total activity rates for the network were stable over the course of the 
recordings. To construct binary states (zero/one) of multiple time series we sampled them at one half the spike rate 
of the most active cell. This gives the highest entropy per measured bit for a temporally uncorrelated data stream. 

III. INFORMATION THEORETICAL ANALYSIS OF NEURONAL SPIKE TRAINS 

Analyzing spike trains to understand how groups of neurons share and process information is challenging because 
neuronal time series typically display strong temporal stochasticity. This is the result of both the spatial sparseness 
of the collected signals of individual neurons, which receive and send processes to many other hidden (not recorded) 
cells, and of the intrinsically non- linear and stochastic nature of neuronal and synaptic dynamics [l|. 



FIG. 1: A neuronal culture (right) over a microelectrode array. The uniform grid of points shows electrodes where neuronal 
activity is measured. Left panels show individual neurons. White bars are 50 /im (center-to-center distances between electrodes 
is 40 fim). 
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FIG. 2: Spike time series for the spontaneous (native) activity of a cortical culture with 62 recorded neurons. Much of the 
activity takes place as part of network- wide collective events (left, vertical features), known as network bursts. The structure 
of each neuron's spike train observed over short time scales (right, network burst detail) shows strong stochasticity. 



The noisy character of neuronal spike trains, see Fig. [2l suggests a probabilistic description of their time series, in 
terms of sequences of active spiking periods and silences [23[ , as strings of zeros and ones. This standard decomposition 
allows the analysis of neuronal time scries as noisy channels via information theory. In particular we can measure how 
the activity of one cell is informed by the state of others and determine their relative configuration, which will allow 
us to reconstruct proxy connectivity networks mapping how information is shared, stored and processed collectively 
in these ensembles. 



A. Mutual information between neurons 



Mutual information provides a natural quantitative measure of the degree of statistical dependence between two 
stochastic variables, free of assumptions about the nature of the underlying joint distribution. As such, mutual 
information is a non-linear generalization of familiar correlation functions [24j |. In the neuronal networks under 
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FIG. 3: Different connectivity arrangements of neurons have distinct roles in short term memory or in information processing. 
These can be identified via the information theoretical analysis of multi-cell spike time series, with the former associated with 
R > and the latter R < 0, see text. 



consideration here, large mutual information between suitably defined time series of two neurons will be taken as an 
indication of connectivity between them. Subtleties that arise with this association will also be discussed and, at least 
partially, resolved below. 

We start by recalling the definition of the mutual information between two stochastic variables XY 

wn-EK..ni«(^). m 

where p(x,y) is the joint distribution and p(x),p(y) are the single variable marginals. The mutual information 
is semi-positive definite, and equal to zero if and only if the two variables X, Y are statistically independent, i.e. 
p(x,y) =p(x)p(y). 

The magnitude of the mutual information between two cells is dependent on the information content of the time 
series of each cell, in terms of their Shannon entropy H(X), H(Y). This becomes clear through consideration of the 
relations 



I(X;Y)=H(X)-H(X\Y)=H(Y)-H(Y\X) — » I(X: Y) < min [H(X), H(Y)] . (2) 
This bound, taken together with I(X;Y) > 0, implies that the normalized mutual information i{X;Y) 

l{X ' Y) = min[H(X),H(Y)} (3) 

is always in the range < i(X;Y) < 1, and can be used to measure the strength of the link between two cells. In 
this spirit, more restricted measures of statistical dependence such as cross-correlation have also been used to identify 
putative links between simultaneously recorded neurons [2fj| . 



B. Multiple cell functional arrangements 



We would like to not only reconstruct a network reflecting putative connectivity between neurons, but also to 
identify how information is relayed or processed through neuronal activity. Fig. [^distinguishes relaying and processing 
computations for arrangements of three neurons. Sequential chains of cells can relay information, and are known to 
play a part in short term memory in the brain [251 | . On the other hand neurons receive inputs from many other 
cells, so that their spiking activity may behave as the (non-linear) function of multiple inputs. In this way individual 
neurons naturally integrate activity from many sources and are able to process information. Here we show how these 
two different types of structures can be distinguished given data streams for multiple cells. Consequently we can 
determine which type of functional structure is more prevalent in cortical cultures. While three-cell networks are still 
extremely small relative to even a single cortical column (approximately 10 5 neurons), they are complex enough to 
illustrate the elementary properties of network computation. 

The computations performed by networks of neurons can be analyzed via the consideration of appropriate multi- 
informations between groups of cells. To see this consider that the notion of mutual information, Eq. |T]), naturally 
generalizes to higher dimensions. For example I{X\, X2, ■ ■ . , X n ) is the mutual information between the joint distri- 
bution of n variables p({xi}) and the product of each single variable's distribution. This quantity is given explicitly 
for n=3, for several examples in table [ pb)| . Similarly we will write I({Xi, X%, . . . , X n }; Y) to denote the mutual 
information between the joint state of n variables {Xi} and variable Y. 
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The use of multi-information between groups of variables is useful because it allows us to examine connectivity 
arrangements involving more than just pairs of elements. In particular, for an arrangement of three elements, we 
consider the conditional mutual information of two of the variables X, Y, subject to the state of a third Z 27], 

I(X; Y\Z) = £ p(x, y, z) log 2 (4) 

I(X;Y\Z) is a mutual information and as such is semi-positive definite and zero only when X, Y are independent 
stochastic variables (each conditioned on Z). Thus I(X;Y\Z) tells us whether consideration of a third variable 
supplies more or less information about the nature of the correlation between X and Y. 
This property suggests the consideration of the difference 

R(X,Y,Z) = I(X;Y)-I(X;Y\Z) (5) 

as a compact means to characterize how knowledge of Z informs us of the state of {X, Y}. Although perhaps not 
immediately obvious this quantity is identical to that recently proposed by Schneidman et al. [2 a , [29( as a measure 
of redundancy of the knowledge of Z given X, Y. In their definition, for three variables 

R(X, Y, Z) = I(X; Z) + I(Y; Z) - I({X, Y}; Z). (6) 

We prove in Appendix A that the two definitions are equivalent, where we also show that R is fully symmetric under 
permutations of its arguments, and consequently can be computed more parsimoniously than its definition may at 
first suggest. 

It is immediately clear from the definition of R that it can be positive, negative or zero. A positive R is equivalent 
to I(X; Y) > I(X;Y\Z) indicating that knowledge of Z does not improve (and may degrade) knowledge of {X, Y} 
correlations. Conversely R < indicates that knowledge of Z leads to more complete information about {X, Y}. 
Finally R — indicates that at least one of the three variables, is independent of the others. In this sense R = 
denotes the absence of a genuine three variable structure. After Schneidman et al. [28| we refer to these three 
situations as redundancy (R > 0), synergy (R < 0) and independence (R = 0). To better convey their meaning we 
illustrate each of these cases with examples. A summary of what can be learned from these quantities about the 
detailed connectivity mapping between variables is given in Table HT1 



1. Independence: R = 

It follows immediately from definition (0| that if Z is independent of {X, Y} then R = 0. Indeed if both X and 
Y are statistically independent of Z, so that p(x\z) — p(x), p(y\z) — p{y) and naturally p(x,y\z) = p(x,y). Then 
I(x, y\z) — I(x. y), which implies R = 0. 

Moreover this is true in general, as expected from symmetry of R, if any of the three variables is independent of 



the other two. We see this for the Boolean functions X and Y, of tables (a) and |(b)[ In these cases we would infer 
the existence of only one link from the only non-zero binary mutual information, and the absence of a three variable 
structure, from R — 0. 



8. Synergy: R < 

If Z is a function of both variables X, Y then knowledge of Z leads to greater information of the joint state of 
{X, Y}. In this case R < 0, and we speak of synergy. 

In order to illustrate this point we follow Schneidman et al. [28] in considering a set of Boolean functions of two 
binary variables X, Y . Although these functions are much too stylized to be realized between neurons, they supply us 



with the simplest useful illustrations. Table (a) shows several of the sixteen possible Boolean functions of two binary 
variables. 

In all the cases of Table [(a)] except for CHAIN to be discussed below, X and Y are independent random variables 
(po = pi = 1/2) and are thus statistically independent so that I(X: Y) = 0. Information theoretic quantities for these 
functions are given in Table |(b)[ 

Considering exclusively the mutual information in Table [(b)] as a starting point we see that we would infer correctly 
that X, Z and Y, Z are connected but X, Y are not for the functions AND, OR (and X H Y and Y H X, not shown). 

We may find it surprising however that for XOR we would infer no connectivity (all pairwise mutual informations 
are zero). We find nevertheless a negative R and a nonzero I(X,Y, Z), which tell us that the three variable joint 
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(a)Examples of the relationships among three Boolean variables, showing 
synergy, independence or redundancy, see text. Nontrivial logical circuits 
(e.g. OR, AND) are examples of functions for which output is a joint 
function of the two inputs and are thus associated with information 
processing. Outputs that reflect only one of the inputs are trivial as three 
cell structures (independence). CHAIN is an example of a third type of 
relation where variables are not independent, but knowledge of any third 
cell does not convey new information. 
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(b)Information theoretic quantities for the Boolean functions of Table [(a)] Inputs 
for X, Y are taken at random so that H(X) = H(Y) = 1, except for CHAIN, where 
x=y, resulting in the only case where there is mutual information between the two 
inputs. The functions YHX and XHY give the same results as AND and were thus 

omitted for simplicity. Note that there are possible connectivity structures, like 
XOR, which have zero mutual information between any two links, but still exist as 

syncrgetic triplets. 



Function 


H(Z) 


I(X;Y) 


I(X;Z) 
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TABLE I: Examples of 3 cell functional arrangements and their information theoretical signatures. 



distribution is non-trivial. Thus we infer a genuine three cell object present, despite the absence of pairwise mutual 
informations. Such structures would be missed under exclusively pairwise analysis. 

Finally we notice that a fully connected diagram, in which variables are not Markovian (see below) is also synergetic 
and therefore implies R < 0. 



3. Redundancy: R > 

Finally we consider cases where knowledge of a third variable Z provides redundant information about to the state 



of {X, Y}. The simplest examples are the functions CHAIN of Table (a) CHAIN can be seen as the limit of the 
(anti)ferromagnetic interaction considered in Schneidman et al. (28j , as the coupling /3 — > oc. 

Clearly the state of the third variable, while not independent, does not add information about the state of the 
other two. This situation is generic of connections organized as chains. An important and tractable situation is when 
variables are Markovian, in which case a well known result, the information processing inequality, emerges. This 
implies in turn that R > 0, generally, so it is worth explicit consideration. 

If X , Y, Z form a Markov chain, denoted X — > Y — > Z, then their joint distribution can be written as [27j 

P{x, y, z) = p(x)p(y\x)p(z\y). (7) 

This means in particular that X, Z are independent given Y , i.e. I(X; Z\Y) = 0. In this situation the information 
processing inequality (2?| also tells tells us that for a Markov chain X — > Y — > Z 

I(X;Y)>I(X;Z). (8) 

Now, writing I({X, Y}; Z) as (see Appendix A) 



I({X, Y}- Z) = I(X; Z) + I(X; Y\Z) = /(A; Y) + I(X; Z\Y) 



(9) 



7 



and using I(X; Z\Y) = we conclude that 

R = I(X;Y)-I(X;Y\Z)=I(X;Z)>0. (10) 

The same result is obtained for the chain Z — > Y — > X, which from the point of view of information quantities is 
equivalent to its reversed order considered above. 

For the chain X — > Z —> Y, for which I(X; Y\Z) = 0, it follows immediately that R = I(X: Y) > 0. In particular 
if the variables are indeed connected then I(X; Y) > strictly and R > is a general property of Markov chains. 

Finally we note that if Z drives both X and Y, as in X <— Z — > y then i? > 0. We may suspect this from the fact 
that Z is then a function of a single variable. Nevertheless formally this follows from the form of the joint distribution 

p(x, y, z) = p(z)p(x\z)p(y\z) => p(x, y\z) = ^ = P(x\z)p(y\z), (11) 

which implies I(X;Y\Z) — and therefore that R = I(X;Y) > 0. In the case of R > we therefore preserve only 
the two strongest links among the three cells, a strategy also adopted in [3(| ■ 

4- Normalization of R 

Just as for the mutual information the absolute value of R is dependent on the magnitude of the various multi- 
informations involved in its definition. We can place upper and lower bounds on R, so that a normalized quantity 
r(X, Y, Z) can be constructed, that factors out these effects. 

First note that it follows directly from the definitions (see also appendix A) that 

R < min [I(X; Y), I(X; Z);I(Y; Z)\ . (12) 
So that if R > 0, r = R/min [I((X; Y),I(X; Z);I(Y; Z)\. Similarly when R < then 

R > -min [I(X; Y\Z),I(X; Z\Y), J(F; Z\X)} . (13) 

Note also that 

R > -min [I({X, Y}; Z),I({X, Z}; Y),I({Y, Z}; X)] , (14) 
but that the first bound is more stringent (see appendix A). Thus for R < a normalized r is given by 

R 



min [I(X; Y\Z), I(X; Z\Y),I(Y; Z\X)\ ' 



(15) 



IV. THE STRUCTURE OF NETWORKS OF SPIKING NEURONS IN VITRO 



In this section we show how to use the information theoretic diagnostics of the previous section to construct 
connectivity graphs for neuronal ensembles. 



A. Graph construction in practice 



The joint consideration of R and of pairwise mutual informations leads to a classification of the connectivity 
diagrams of any three nodes. The summary of these results, which can be implemented as an algorithm to map the 
network connectivity, is summarized in Table ITT] An efficient algorithm computes first the value of R for a chosen 
triplet of variables. This determines how many links to attribute. The strength of these links, whenever appropriate, 
are determined by mutual information between pairs of variables. Whenever the consideration of several trios implies 
different weights for the same connection between two neurons, the results are averaged. 

To determine whether measured values of R and / are significant, we compute their values according to a null model 
where every cell i in the network produces spikes at the same average rate as observed for each neuron's actual spike 
train, but whose temporal structure is otherwise random. This corresponds to a number of spikes per unit time given 
by a Poisson distribution with rate or equivalently, as we do in practice, a time interval between consecutive spikes 
given by an exponential distribution. Under perfect estimation this Poisson model generates statistically independent 
spike trains between any two cells. In practice estimation is imperfect and this procedure leads instead to very small, 
but non-zero, values of R and /. Thus for each trio of cells we take the observed R to be significant if it is larger 
than three times that measured via the Poisson model. Otherwise the trio is considered statistically independent with 
R = 0. We proceed similarly for the mutual information measured for any two cells. 
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No connections: 


Vi,j£{l,2,3}I(Xi,Xj) = 




1 connection: 


3! !je{ i, 2 ,3}/(^,^)/0 






There is a 3-cell connected structure 




2 connections: 


R > (redundancy) 
X^Y^Z X^rZ^Y X^Z^Y 
Markov: Markov: 
I(X; Y) > I(X; Z) I(X; Z) > I{X; Y) 
I(X;Z\Y) = I(X;Y\Z)=0 I(X;Y\Z)=0 
R = I(X;Z) R = I(X;Y) R = I(X;Y) 




3 connections: 


R < (synergy) 
X -» Z «- Y: Z = f(X,Y) 
Each variable is a function of all others. 



TABLE II: The joint consideration of the values of R and of the binary mutual informations I lead to a classification of the 
connectivity arrangements between three stochastic channels. For R > only the two strongest links are adopted. For R < 
we adopt three links as a general representation of the specific functional interdependence between the three cells. 



B. Graph Theoretic Analysis 

Next we characterize the structural properties of the resulting networks in terms of graph theoretic quantities. 
Although all three graphs are small in terms of numbers of nodes we show that results about their network structure 
are consistent for the different networks. Figure |4] shows the weighted graphs for two of these networks. Figure [5] 
shows the connectivity weight distribution and the histogram of r for all cell trios. 

The distribution of connectivity weights is well-described by an exponential. This implies that most links are weak, 
with a few stronger ones tending to inter-connect subsets of cells. The distribution of r shows that most trios are 
associated with redundant structures that may be involved in short term memory functions. Nevertheless a number 
of trios display r < 0, indicating the existence of non-trivial functional relations between measured cells, that may 
play a role in information processing. 

To compare the structure of these weighted graphs quantitatively to other complex networks we transform them 
into binary graphs, with each link set to zero or one if its strength lies below or above a given threshold, respectively. 
Table HIT1 shows the average clustering coefficient, diameter and Pearson coefficient for degree-degree correlations for 
the three networks as functions of the threshold, and compares them to results from one hundred realizations of the 
standard null model, viz. a network with the same node degrees but with randomized connections. 

We see that in every case the neuronal graphs display significantly higher clustering than the null networks. They 
also show small diameters of the largest component d, of order d ~ ln(iV), where N is the number of nodes. These 
two properties together indicate that cortical networks in vitro, like the neuronal network of the nematode Q, are 
small world graphs. 

Finally we observe that cortical networks in vitro are weakly disassortative, i.e. that the Pearson coefficient associ- 
ated with degree-degree correlations is, in most cases, negative but small (3f[. This indicates that nodes of dissimilar 
degree preferentially connect to each other. Together these measures show that networks of neurons, constructed via 
the information shared by groups of cells, are significantly different from randomized graphs, but share instead many 
structural properties with other complex networks [H, [H, [|| . 



V. DISCUSSION AND CONCLUSIONS 



The understanding of the functional network structures used by living nervous systems to store and process infor- 
mation is key to unravelling the mechanisms of their computational power. It is now possible to measure the activity 
of networks of individual neurons and to start to identify such informational structures, their development over time 
and their role in response to external stimuli (see e.g |32[). Here we have shown how time series for networks of 
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FIG. 4: Weighted graphs for network of a) 62 and c) 33 recorded neurons. Panels b) and d) show the corresponding networks 
selecting only the links with strength above 0.4. The graphs are characterized by a large number of weak links that connect 
almost every pair of cells, and of fewer stronger links that tend to interconnect subsets of neurons. Node size is drawn 
proportionally to its total connectivity weigh. 



Size 


Threshold 


Av. 


Clustering 


Diameter 


Assort at ivity 


(recorded neurons N) 


[0,1] 


Actual 


Random 


Actual 


Random 


Pearson coeff. 




0.1 


0.89 


0.52± 0.04 


2 


2.97± 0.03 


-0.150 ± 0.025 


N=20 


0.3 


0.57 


0.24 ± 0.06 


4 


4.15 ± 0.55 


-0.040 ± 0.048 




0.5 


0.32 


0.05 ± 0.10 


1 


2.10 ± 0.70 


-0.200 ± 0.100 




0.1 


0.83 


0.48± 0.02 


3 


2.97± 0.03 


-0.114 ± 0.001 


N=33 


0.3 


0.61 


0.26 ± 0.04 


4 


3.99 ± 0.30 


-0.033 ± 0.026 




0.5 


0.07 


0.00 ± 0.02 


2 


2.98 ± 0.86 


0.064 ± 0.053 




0.1 


0.82 


0.51± 0.01 


3 


2.97± 0.03 


-0.025 ± 0.004 


N=62 


0.3 


0.60 


0.37 ± 0.02 


4 


4.00 ± 0.20 


-0.093 ± 0.004 




0.5 


0.43 


0.24 ± 0.02 


3 


3.93 ± 0.18 


-0.102 ± 0.006 



TABLE III: Graph theoretic measures for each of three cortical neuronal networks and as functions of the weight threshold, 
and their comparisons with the average and standard deviation obtained from a graph with the same degree but randomized 
connections. Cultured neuronal networks are clearly distinguishable from randomized networks with the same degree. They 
are small world graphs with weakly disassortative degree correlations. 

spiking neurons can be treated statistically and analyzed in terms of informational theoretic quantities to reveal how 
information is shared, relayed and processed by groups of cells. This analysis generates networks of shared informa- 
tion, which are proxies for the physical connectivity between neurons. In fact, knowledge of this type of connectivity 
and its dynamics may well suffice to reveal the underpinnings of collective computation in networks of neurons. 

We have also shown that consideration of information theoretic quantities beyond two cells can identify structures 
that are naturally higher order, revealing for example a cell whose output is a non-linear function of several others. 
The specificity of such structures is lost when we represent them as sets of binary links, as we have done here to 
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0.2 0.4 0.6 0.8 1 -1 -0.5 0.5 1 



Weights r 

FIG. 5: The probability distribution for weights of the inferred network of 62 recorded cells (left). This distribution is well 
described by an exponential (dashed line), with average connectivity weight 0.294 ±0.019. The distribution of normalized r for 
the same network (right). Most trios show information redundancy (r > 0), while a smaller number display synergy (r < 0). 
The central peak at r — 0, with probability p — 0.091, consists of a fraction of 70% of truly independent trios (R = 0), 22% 
with negative r > —0.05 and 8% with positive r < 0.05. The other two networks exhibit quantitatively consistent results, but 
with poorer statistics. These results are suggestive that multi neuron connectivity structures associated with both information 
processing (R < 0) and short term memory (R > 0) are present in cultured neuronal networks in vitro. 



produce standard network analyses. The systematic mapping and characterization of these functional modules goes 
beyond the scope of the present work, and will be pursued elsewhere. 

Neglecting such functional specificity we have built networks of shared information, which we showed to be weakly 
disassortative small world graphs, in agreement with e.g. the detailed neuronal network of C. elegans 0, but in 
contrast to networks of correlation between large scale brain regions, identified via fMRI measurements 7J. The 
distribution of weights is approximately exponential and multi neuron structures associated with information relaying 
dominate over a smaller number of cell arrangements associated with information processing. Regardless of details of 
specific proportions for these cellular arrangements we showed, by identifying multi-neuron structures with R > and 
R < 0, that neuronal modules associated with short term memory and information processing co-exist in disassociate 
neuronal cultures in vitro. How these structures and their generalizations to larger cell numbers may be harnessed to 
perform reliable computation remains perhaps the clearest single challenge for future studies of information processing 
in neuronal networks living in vitro. 

These results raise several other interesting questions for future research. It is natural to expand the analysis 
presented here to include multi- information quantities involving more cells, in order to identify the functional role 
of more complex neuronal arrangements. It is also interesting to investigate their change over time, and under 
electrophysiological stimulation. Finally, this type of analysis can be applied to any distributed information processing 
system, natural or artificial, and should prove useful in revealing the computational structures that constitute other 
types of nervous tissue, in vivo and in vitro, and in creating mathematical models and engineered systems that mimic 
them. 
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Appendix A 

In this appendix we prove the equivalence of the definitions of R, Eqs. (O and ©. This equality establishes that 
the measure R of redundancy, proposed by Schneidman et al. in Ref . [281 ] , can be written, for three variables in terms 
of the difference between the mutual information of a pair of variables and its mutual information conditional on a 
third cell ©. 
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We begin with the definition of R given two variables X, Y, and upon consideration of a third Z, which is a candidate 
for having connections with X, Y or may be an external stimulus 

R = I(X;Z)+I(Y;Z)-I({X,Y};Z), (16) 

where, as usual, the semicolon separates the states being compared, so that the last mutual information is between 
the joint state of {X, Y} and Z. We will now prove that R, defined in (p~6|) can also be written as 

R = I(X;Y)-I(X;Y\Z). (17) 

Starting from (|16p we use the chain rule for mutual information [27j 

n 

I{{X U X 2 , X n }; Z) = J2 T ( X i> ZlX^Xi.2, . . . , X x ). (18) 

i=l 

Specifically for three variables (n=2) 

I({X, Y}; Z) = I(X; Z) + I(Y; Z\X) = I(Y; Z) + I(X; Z\Y), (19) 
I({Y, Z}; X) = I(X; Y) + I(X; Z\Y) = I(X; Z) + I(X; Y\Z), (20) 
I({X, Z}; Y) = I(Y; Z) + I(X; Y\Z) = I(X; Y) + I(Y; Z\X). (21) 

These expression can now be used in (|16j) to give 

I(X; Z) + I(Y; Z) - I({X, Y}; Z) = I(Y; Z) - I(Y; Z\X) = I(X: Z) - I(X: Z\Y). (22) 

Similarly we can write 

I(X; Y) + I(Z; Y) - I({X, Z}; Y) = I(X; Y) - I(X; Y\Z) = I(Y; Z) - I(Y; Z\X), (23) 
I(Y; X) + I(Z; X) - I({Y, Z}; X) = I(Y; X) - I(Y;X\Z) = I(Z; X) - I(Z;X\Y). (24) 

These identities taken together show that the definition of R is symmetric under variable permutations and in 
particular proves our proposition 

R = I(X; Z) + I(Y; Z) - I({X, Y}; Z) = I(X; Y) - I(X; Y\Z). (25) 

As a corollary we see that R is fully symmetric, i.e. 

R = I(X; Z) + I(Y; Z) - I({X, Y}; Z) = I(X; Y) + I(Z; Y) - I({X, Z}; Y) 

= I(Y;X)+I(X-Z)-I({Y,Z};X) 
= I(X: Y) - I(X: Y\Z) = I(X; Z) - I(X; Z\Y) = I(Y: Z) - I(Y; Z\X). (26) 

Thus R can be computed efficiently through the most convenient set of estimated distributions. 

In the case of more than three variables analogous expression for R in terms of conditional mutual informations can 
be written easily, through (|18[) . but become less elegant as the distributions are now conditional on several variables. 
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